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ABSTRACT 

This thesis traces the development of the hierarchical, dynami- 
cally reconf igurable , input/output network which has been constructed 
at the Draper Laboratory. It presents a summary of the design pro- 
cedures used in determining the network architecture, communication 
methods, message formats, and overall topology. Further, it describes 
both the hardware and software features that have been implemented in 
the network's microprocessor-based nodes. In addition, the centrally 
located software algorithms developed to configure, repair, and monitor 
the network have been extensively discussed. Finally, the thesis also 
includes a reliability analysis of the network in a typical application, 
and a performance evaluation of the effectiveness of the configuration 
and control programs. 

The implementation of a hierarchical, dynamically reconf igurable 
network is a radical departure from the typical data bus oriented I/O 
systems found in many applications today. The justification for this 
departure lies in the improved damage - and fault-tolerant features, 
not found in other architectures, that the network possesses. Specifi- 
cally, through the alternate path redundancy provided by the inactive 
links, the centralized controlling element is capable of dynamically 
reconfiguring the surviving portions of a damaged network in order to 
circumvent the malfunctioning elements. Thus, the overall reliability 
of the I/O system has been improved, since it is now possible to main- 
tain communication with each peripheral node and its host processor, in 
spite of the occurrence of moderate levels of physical damage. 
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Two variations of the basic network design have been developed. 
One, termed the single level network, is the standard form of the I/O 
net used in conjunction with the Laboratory* s OSIRIS (Onboard Surviva- 
ble Integrated Redundant Information System) demonstration implementa- 
tion of a commercial aircraft flight control system. All nodes in the 
single level network are on the same hierarchical level and consequently 
communicate in an identical manner with the central computer. The 
second variation of the basic design is called the bilevel network. In 
this case, two separate hierarchical levels exist independently, joined 
at only one point of tangency, the bilevel node. The advantage of the 
bilevel network over the single level network arises in applications 
where the computational load is great at the various local processors. 
Since the bilevel network is able to effectively isolate the computa- 
tionally intensive nodes in a lower level network, not in direct com- 
munication with the central processing element, an increased processor 
throughput is potentially possible. 

To date, experimentation with a six-node single level network 
has indicated that the percentage of the available I/O bandwidth re- 
quired for network management functions is compatible with the operation 
of the digital autopilot application program. Additionally, the average 
time required to detect a fault, load the software reconfiguration task, 
and correct the indicated malfunction, given the characteristics of the 
hardware currently implemented, is on the order of 1 sec. Finally, the 
software overhead in the central computer for the network control 
programs has amounted to less than one thousand sixteen bit words, plus 
an added three hundred words for system tables and constants. Overall, 
the hierarchical, dynamically reconf igurable I/O network, is conceptual- 
ly well suited to the broad range of applications requiring a high 
availability input/output system. 

Thesis Supervisor: Albert L. Hopkins, Jr. 

Title: Staff Member, The Charles Stark Draper Laboratory, Inc. 

Thesis Supervisor: Hoo-min D. Toong 

Title: Assistant Professor of Electrical Engineering. 
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CHAPTER 1 



INTRODUCTION 



The potential role of distributed processing has become increas- 
ingly important in the real time control and data management environ- 
ments of many commercial, industrial, and military systems. Because 
of the declining costs of hardware, the placement of a significant 
amount of computational capability at a remote location has become 
feasible. However, the inherent physical separations, and prolifera- 
tion of additional sub-systems resulting from such a distribution, have 
resulted in a dramatic increase in the volume of input/output (I/O) 
communications required. Clearly, it is essential that this added 
communications load must be made as reliable as possible in order to 
satisfy the stringent performance requirements imposed by most high- 
availability systems [24] . 

To achieve highly reliable inter-device communications, some 
type of fault-and damage-tolerant implementation is normally needed. 
Taking the form of redundant hardware, alternate communication paths, 
or error recovery procedures, for example, these features provide a 
distributed I/O communications system with the capability for "graceful 
degradation" [3] [16]. In other words, the system can continue to 

operate correctly, even in the presence of a predetermined threshold 
of hardware or software faults, or after incurring moderate levels of 
physical damage. Many such I/O systems exhibiting a wide range of 
fault- tolerant capabilities have been built during the past several 
years [24] . This thesis will trace the development of one such imple- 
mentation for a commercial aircraft flight control system that has been 
constructed at the Charles Stark Draper Laboratory. 

Since the late 1960's, the Draper Laboratory has been investi- 
gating the application of fault-tolerant multiprocessing technology to 
a variety of digital control applications, primarily to the area of 
aircraft flight control systems [16] [25] [29] . Out of this continuing 

investigation has emerged the OSIRIS (Onboard Survivable Integrated 
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Redundant Information System) concept. OSIRIS is a real-time distribu- 
ted processing system consisting of one or more fault-tolerant multi- 
processors , a damage-and fault-tolerant network, physically separated 
local processors, and operational software for fault detection, identi- 
fication, and recovery [15] (see Figure 1.1). An experimental version 
of OSIRIS is presently in existence, and it is the OSIRIS input/output 
network which is to be studied in this thesis. 

The OSIRIS fault-and damage-tolerant network, first envisioned 
in 1973 [29], was originally implemented in 1974 as a six-node simplex 
network of hardwired circuit switches under central software control 
[25] . This demonstration was evaluated under an Office of Naval Re- 
search contract to investigate the possible application of the network 
approach to shipboard data management systems. Soon after this origi- 
nal effort had been successfully completed, the rapid rise of low cost 
microprocessors signalled that a more flexible follow-on implementation 
could be realized by centering the design of the network node around a 
microprocessor. Consequently, work was initiated in early 1975 to in- 
corporate the microprocessor into the demonstration network. At the 
same time, it was decided to utilize as the central processing center 
for the network the CARDS multiprocessor, another C..S. Draper research 
effort. With this decision the "breadboard” OSIRIS system had begun to 
take shape. The central processing center, or CPC, possessed an exten- 
sive assortment of fault-and damage-tolerant features. It was able to 
execute, simultaneously, multiple tasks in addition to the software re- 
quired for the I/O network configuration and control. Some of the more 
significant features of the CARDS multiprocessor are as listed below: 

1. triply modular redundant processors 

2. triply modular redundant memories 

3. triply modular redundant I/O modules 

4 . triply redundant serial I/O bus 

5. dual redundant line drivers and receivers 

6. additionally, each I/O module contained among other things, 
bus isolation gates and error detection and recording cir- 
cuitry . 

Subsequent to the beginning of work on the network of micropro- 
cessor nodes, a follow-on to the previously cited Office of Naval Re- 
search contract was received. The objective of these additional funds 
was to demonstrate a concept, proposed earlier in reference [26], known 
as a "bilevel network". This concept evolved as an attempt to more 
fully realize the hierarchical system potential of the general 
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Figure 1*1 Block Diagram of Demonstration OSIRIS System* 
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network architecture [26], In a bilevel network, certain specialized 
nodes, known as "bilevel nodes", would allow two distinct smaller net- 
works to exist independently, joined only at a single point of tangency, 
the bilevel node. If the two networks were hierarchically related (i.e. 
one of them subordinate to the other) then the bilevel network could 
effect a clear division of the network control versus computational 
responsibilities. This characteristic/ potentially, could result in a 
significant improvement in local processing throughput, when compared 
to the results obtainable from the simple or single level network 
already in development. Through fairly modest additions to the hard- 
ware for the microprocessor node, it was determined that a bilevel 
node could also be constructed. 

Thus, as this thesis began, work was underway, not only on the 
incorporation of the microprocessor into the fault-and damage-tolerant 
input/output network of the demonstration OSIRIS system, but on the 
preliminary stages of the eventual modification of the single level 
network into a bilevel network. It is the objective of this effort 
to contribute to both of these research goals stated above, in the 
following manner: 

1. To trace the design processes used in both the single and 
bilevel networks. 

2. To describe the hardware and nodal operating systems that 
have been developed by C.S. Draper personnel [28]. 

3. To develop, implement, and evaluate the software 
algorithms needed to control and configure both the single 
and bilevel networks. 

4. To present some indication of where the current network re- 
search effort should proceed next, based on the progress 
made to date. 

Although the approach to a fault-and damage-tolerant I/O network 
as implemented in the OSIRIS system is unique, other existing computer 
systems do utilize, to varying degrees, fault-tolerant features in 
their inter-communication schemes. Besides the normal error detection 
and correction performed on incoming data by most systems, networks 
such as the ARPA (Advanced Research Projects Agency) network also 
possess a reconfiguration capability [6]. This feature is made pos- 
sible through the use of its packet switching node computers known as 
IMP*s (Interface Message Processors). In the absence of normal traffic. 
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each IMP transmits idle packets on unused lines at half second inter- 
vals. The lack of a return packet or incoming traffic indicates a 
faulty line and allows the dynamic routing tables at each node to be 
updated accordingly. Thus, faulty links are bypassed and previously 
spare links are activated automatically [23] . Though not as complex, 
CYBERNET, of the Control Data Corporation, also has a fault-detection 
and correction capability, but it requires human intervention to acti- 
vate the redundant links [17] . Still other telecommunications networks 
such as MERIT at the University of Michigan and OCTOPUS at Lawrence Liver- 
more Laboratories in California also implement redundant links to pro- 
vide alternate communication paths in the event of link or processor 
failures [7] . In all the examples given, the need for a highly reliable 
intercommunications system is satisfied to a great extent by utilizing 
the dual concepts of redundancy and reconfigurability. These two fea- 
tures are also found to be dominant in the design of the OSIRIS input/ 
output network. 

In the chapters to follow, the development of both the single 
and bilevel networks is examined. Chapter 2 discusses the various 
decisions that were made during the preliminary stages of the network 
design. Chief among these include: 

1. I/O architecture selection 

2. Communications method selection 

3. Link and data characteristics 



4. Topology selection 

Chapter 3 expands, in fairly great detail, upon the features of the 
single level six-node network. Included in its discussion are the 
following topics: 



1. General description and capabilities 

2. Demonstration topology selected 

3. Nodal hardware 



4. Nodal operating system 

5. Network configuration and control software 

6. Incorporation of network control and configuration software 
into the CPC 



Chapter 4 repeats an identical development of the bilevel ten-node 
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network. Unfortunately, the complete hardware implementation of the 
ten-node network has not yet been completed. Chapter 5 describes the 
general problem of evaluating the reliability of a typical OSIRIS I/O 
network. It also discusses, assuming a number of apriori conditions, 
the increased reliability characteristics that the alternate paths in 
the six-node network contribute. Chapter 6 treats the subject of the 
performance of the single level network, particularly in the area of the 
configuration and control algorithms 1 evaluation. Finally Chapters 7 
and 8 present topics for future investigations, and thesis conclu- 
sions , respectively . 



16 



CHAPTER 2 



FAULT-TOLERANT NETWORK DESIGN ALTERNATIVES 



2 . 1 Design Constraints Imposed by the Airborne Environment 

The initial step in the development of any general communications 
network is to identify the set of parameters over which the design is 
to be a function. Among the more important of these items are the 



following 


[20] : 




1 . 


Total cost 




2. 


System reaction time 




3. 


Network survivability and vulnerability considerations 


4. 


Network efficiency 




5. 


Network user requirements 




6. 


Serial or parallel transmission 




7. 


Circuit-switched or packet-switched procedures 




8. 


Message routing procedures 




9. 


Network control 




10. 


Security 




11. 


Network topology 




For the OSIRIS input/output network, several of the above design 
considerations, when placed in the context of the airborne environment, 
form a set of constraints over which the network design must be 
optimized. Specifically, the following are the important questions 
to be answered during the network design process: 


1 . 


Total Cost 





How can the desired levels of performance and reliability 
be achieved, while still minimizing the total cost, and often 
more importantly, the total weight of the system? 
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2 . System Reaction Time 



Can the functions of network configuration and control be 
accomplished in a relatively small percentage of the total I/O 
bandwidth? If not, their utilization will interfere with the 
primary flight control functions of the overall system, thereby 
degrading the system reaction time. 

3 . Network Survivability and Vulnerability Considerations 

What is the desired level of reliability required? How 
is it defined? How much tolerance to faults and physical damage 
should be incorporated? How best should these survivability 
features be implemented? 

4 . Serial or Parallel Transmission 

What method of communication is best suited to the net- 
work? How is it implemented? What do the actual data links 
look like? Is the communications method chosen compatible with 
the I/O bus of the CPC? 

5 . Circuit-Switched or Packet-Switched Procedures 

How does the hierarchical architecture affect the selec- 
tion of nodal switching procedures? How does transmission de- 
lay affect both methods? Is there an application where both 
alternatives could be utilized? 

6 . Network Control 

What types of transmissions should be developed to con- 
trol the network? What is necessary to effectively configure a 
network, verify its correct operation, and reconfigure it when 
a fault is detected? 

7 . Network Topology 

Is there an optimum arrangement of data links for the 
network given the geographic locations of the nodes and the 
number of I/O ports per node? Over what characteristics is 
this optimization process to be conducted? How sensitive is 
the network performance to variations in topology? 

Answers to this extensive list of important questions will be 
presented, not only in subsequent sections of this chapter, but also, 
to a varying extent, in all succeeding chapters. Throughout the devel- 
opment; however, a greater emphasis will be placed upon achieving the 
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required fault-and damage-tolerant capabilities, rather than upon 
minimizing such considerations as total cost, software simplicity, or 
the total weight of the linkage mass. 

2 • 2 Selection of the Network Architecture Over Other I/O Alternatives 

The decision to utilize a network scheme for the OSIRIS input/ 
output communications method was based on a careful examination of the 
advantages and disadvantages of the various fundamental fault-tolerant 
communications architectures. This comparison, as summarized in Table 
1, served to differentiate between the three most common structures; 
the dedicated or star connection, the redundant bus connection, and the 
network connection (see Figure 2.1). Though each alternative possesses 
definite strengths and weaknessess, the overriding concern in the OSIRIS 
application of providing an uninterrupted data stream to the central 
processing center, tended to narrow the spectrum of acceptable I/O 
alternatives. Specifically, the communications structure to be selected 
must, among other capabilities, be able to continue to function correct- 
ly in the presence of moderate levels of physical damage [15] . Addition- 
ally, it is desired that the occurance of isolated faults in one node 
have a minimal effect on other nodes. In other words, the faults must 
be uncorrelated in order that the validity of the data stream not be 
degraded [16] . With these restrictions and the comparisons of Table 1 
in mind, the network architecture was chosen as the logical choice for 
the OSIRIS I/O system. 

In essence, the selection of the network structure for the OSIRIS 
system was a compromise between the dedicated and redundant bus archi- 
tectures. Like the redundant bus, a network has a large linkage mass 
resulting from the necessity of providing alternate communication paths. 
Yet, like the dedicated connection architecture, it does not require 
complete replication in order to achieve routing redundancy [25] . As 
an additional consideration, in a hierarchical design the network con- 
trol intelligence is centrally located, often in a well-protected, 
redundant, highly damage-resistant environment. This is a key feature, 
for as long as the central processing element remains functional, the 
network as a whole is able to survive in the face of a partial loss of 
capability. Through its ability to reconfigure the remaining network 
connections, the CPC can isolate the damaged portion of the network and 
effectively circumvent it in most cases. Another noteworthy advantage 
of the hierarchical structure is found in the one-way initiation of all 
communications by the CPC. Since the central processing element is the 
top level of control in the system, all data, control, and configuration 
requests are originated in the CPC, thereby greatly simplifying the 
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DEDICATED (STAR) 




REDUNDANT BUS 




NETWORK (TREE) 




Figure 2.1 Data Communication Structures. 



20 



TABLE 1 

COMPARISON OF MAJOR FAULT- TOLERANT 
COMMUNICATION STRUCTURES 



ADVANTAGES DISADVANTAGES 

1 . Dedicated Connections (STAR Network) 



Simplest to implement. 

Failure of one node does not 
"bring-down" entire system (in- 
herent "graceful degradation") . 



2 . Redundant Bus Connections 

Fewer links and less cable 
weight than dedicated 
connections . 

Has good growth potential. 

Less complex than network (mini- 
mal operational and reconfigu- 
ration software overhead) . 

Most widely used (less develop- 
mental risk) . 

3 . Network Connection 

. Greatest tolerance to physical 
damage (reconfiguration) . 

. Simplifies the implementation of . 
hierarchical processing systems. 

. Centrally located network control. 

Simplifies fault identification. 



Difficult to expand once central 
computer enclosure has been built. 

Requires excessive wire weight 
and bulkhead penetrations. 

Communication to node lost if 
dedicated link fails. 

Underutilizes the connection 
medium and interface electronics. 

More vulnerable to damage than 
network . 

Not readily adaptable to point- 
to-point fiber optic implementa- 
tion (bus coupler design) . 

Susceptible to "multiplier 
phenomenon" (single failure dis- 
abling a large portion of I/O 
system) . 



Costly in associated node hard- 
ware and software overhead. 

More bulkhead penetrations and 
wire weight than nonredundant 
single bus system. 

Less generally accepted due to 
complexity of configuration 
control . 
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message routing procedures [29]. Stated differently, no provisions 
need be made in the communications protocol to allow for node to node 
inter-communications, or to provide for transmissions to the CPC initia- 
ted by a node. In summary, although the redundant bus and dedicated 
connection structures are excellent I/O architectures for many classes 
of applications, the requirements put forth by the OSIRIS system dic- 
tate the use of a hierarchical network for its I/O scheme. In a related 
application, the network approach is also well-suited to a naval ship- 
board environment. In this case, many of the same restrictions aimed 
at insuring high levels of performance and reliability that apply in 
an airborne application, are also critical to the shipboard command 
and control function [17] . 

2 . 3 Serial or Parallel Transmission 

Three considerations formed the basis for the selection of the 
transmission method for the OSIRIS I/O network. 

1. Will the method be feasible in a widely distributed connec- 
tion scheme involving several nodes and moderately long 
links? 

2. How does the ease of implementation of the hardware communi- 
cations interfaces compare for the two alternatives? 

3. Is the method compatible with the internal I/O bus of the 
CARDS multiprocessor to which it will be attached? 

On the basis of these three concerns, it was decided to use serial 
asynchronous transmission throughout the network. Specifically, this 
was the only one of the two alternatives which was both easy to imple- 
ment, and compatible with the central processing centers I/O bus. In 
addition, opting for serial instead of parallel communications 
interfaces reduced the total linkage mass requirements by a ratio of 
approximately eight to one, a significant savings in cost for any net- 
work . 

In the breadboard OSIRIS system standard microprocessor 
asynchronous interface adapters (ACIA's) were chosen as the hardware 
element to be the transmission controllers [ 26 ] . The standard RS-232 
[18] 60 mA current loop was selected as the basis on which the ACIA's 
would transmit a non-return-to-zero (NRZ) type code. Although asyn- 
chronous operation does require additional start, stop and parity bits 
to be sent with each byte transmitted, the resulting reduction in the 
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effective throughput is not significant in the current implementation. 
Still, for any follow-on version of the OSIRIS system, a much greater 
I/O bandwidth will be required. This need will be satisfied, most 
likely, not by converting to a synchronous communications scheme, but 
by operating the ACIA's with faster microprocessors and associated memo- 
ries. In this way, the problems involved in a byte synchronous system 
of transmitting and interpreting correctly the sync signals, can be 
avoided [ 23 ] . 

As far as the compatibility of the network transmission method 
with the CPC's I/O bus was concerned, the decision to utilize serial 
asynchronous transmission was predetermined by the existing bus archi- 
tecture. The internal bus of the CARDS multiprocessor is a triply re- 
dundant serial I/O bus. To interface it with the network, devices 
known as I/O access units (IOA's) have been constructed [16] (see Figure 
2.2). The IOA's contain a voting mechanism to convert the triply re- 
dundant data of the internal bus into a single majority signal [28]. 

The resulting signal is then routed to an ACIA where the requisite 
start, stop, and parity bits are appended before transmission to the 
network. The process is reversed for incoming data from a particular 
node. A copy of the returning signal is placed on each I/O bus so that 
it can be distributed to the applicable ACIA' s in the CPC. 

In conclusion, the decision to implement serial asynchronous 
transmission was made primarily because of the structure of the CPC's 
triply redundant I/O bus, the ease of implementation using readily 
available ACIA's, and the great savings in wire weight when compared to 
that necessary in a parallel system. Furthermore, the I/O bandwidth 
required by the OSIRIS flight control system was not high enough to 
warrant the added expense of either byte synchronous or parallel 
communications . 

2 . 4 Circuit-Switched or Packet-Switched Procedures 

An important determination in the overall design of the OSIRIS 
I/O system was whether circuit-switching or packet-switching pro- 
cedures should be utilized. The characteristics of both switching 
methods are delineated in Table 2. In essence, circuit-switching in- 
volves two operations. First, a communications path must be established 
between the sender and receiver. Once established, the information 
transfer is then allowed to take place. On the other hand, packet 
switching is one example of a common technique known as "store-and- 
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Fault-Tolerant Multiprocessor 
(CARDS) 




Figure 2.2 OSIRIS I/O Bus Block Diagram. 
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forward" [11] . In this method a message is stored at intermediate nodes 
as it makes its way toward its destination. Each time the message is 
forwarded correctly, the previous node is freed from any further re- 
sponsibility for the message upon receipt of a positive acknowledgement. 
Additionally, in a store-and-f orward system some sort of routing stra- 
tegy is required at the node itself, in order to select the next inter- 
mediate node to which the message will be forwarded. In a full scale 
packet-switching network, the ARPA net [6] for example, the bandwidth 
of the various channels is more effectively utilized since the routing 
strategy can be a dynamic function of such parameters as the I/O traf- 
fic load [11]. 

Returning to the OSIRIS system, the decision as to which switching 
scheme to implement in the microprocessor nodes, as has been done 
throughout the design process, was based upon the intended application 
environment of the network. For the single-level I/O network, circuit- 
switching was selected due to its relative simplicity of implementation, 
and its characteristic of negligible transmission delays. This 
is attributable to the fact that once a network is configured, it 
is essentially a bus structure over which messages are 
sent and received, with no intermediate processing by nodes in the 
transmission path. For the bilevel network, however, a combination of 
circuit-switching and packet-switching was neccessary. This stemmed 
from the requirement to be able to communicate between the two levels 
of the hierarchy. To send a message from the central processing 
center to a node in the lower level of the network, the desired trans- 
mission is circuit-switched through the upper level network to a par- 
ticular bilevel node. Since the original message is directed specific- 
ally to that bilevel node, the node stores the entire message in its 
input buffer. Upon interpretation of the message, the bilevel node 
decides that data is required from one of the nodes subordinate to 
it in the lower level. The bilevel node then reformats the message 
and forwards it to the actual destination node. The subsequent re- 
turning data is routed back to the CPC in a reverse manner. The bi- 
level node handles the packet-switching procedures identically to a 
larger scale network’s node except for four basic differences: 

1. It has no dynamic routing capacity 

2. It cannot queue messages 

3. Its buffer lengths are relatively limited 
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TABLE 2 

COMPARISON OF CIRCUIT-SWITCHING 
AND PACKET-SWITCHING 



MAIN CHARACTERISTICS OF 


MAIN CHARACTERISTICS OF 


CIRCUIT-SWITCHED SYSTEMS 


PACKET-SWITCHED SYSTEMS 


1. Logical equivalent of a wire 

circuit connecting the source 
and destination. 


1. No direct connection established. 


2. Real time capability, negligi- 
ble transmission delay. 


2. Real time capabilities limited 
by inherent retransmission delays 


3. Messages are not buffered. 


3. Messages are buffered. 


4. Hardware switches with minimal 


4. Hardware switches with moderate 


intelligence required. 


switching computer required. 


5. Message routing established 
prior to transmission. 


5. Dynamic routing possible; how- 
ever, some packets can become 
lost during message routing. 


6. Any length transmission 


6. Lengthy transmissions are chopped 


permitted . 


into short packets. 


7. Does not allow for transmission 


7. Buffering allows for speed or 


rate or code conversion. 


code conversion. 


8. Fixed bandwidth transmission. 


8. Variable bandwidth according to 
need . 


9. Explicit messages. 


9. Delegation of authority possible. 
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